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(54) Method for detecting skin color in a digital image 



(57) A method for detecting skin color in a digital im- 
age having pixels in an RGB color space generally in- 
cludes the steps of performing statistical analysis of the 
digital color image to determine the mean RGB color val- 
ues; then, if the mean value of any one of the colors is 
below a predetermined threshold, applying a transfor- 
mation to the digital image to move skin colors in the 
image toward a predetermined region of the color 
space; and employing the transformed space to locate 
the skin color pixels in the digital color image. More spe- 
cifically, if the mean value of any one of the colors is 
below a predetermined threshold, a non-linear transfor- 



mation is applied to the digital image to move skin colors 
in the image toward a predetermined region of the color 
space. Then, depending on the preceding step, either 
the digital image or the transformed digital image is con- 
verted from the RGB space to a generalized RGB space 
to produce a gRGB digital image; skin color pixels are 
detected within the gRGB digital image; a first skin color 
image mask is formed based on the detected skin color 
pixels; a masked gRGB image is generated using the 
first skin color image mask; and finally the skin color im- 
age mask is employed to locate the skin color pixels in 
the digital color image. 
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Description 

FIELD OF THE INVENTION 

5 [0001] The present invention relates to digital image processing methods, and more particularly to such methods for 
detecting skin color in a digital image. 

BACKGROUND OF THE INVENTION 

10 [0002] In digital image processing it is often useful to detect the areas in the image that are skin color. This information 
is used, for example, to adjust the skin colors in the image to be pleasing. The location of skin color is also used in 
face detection and recognition algorithms, automatic image retrieval algorithms, and red-eye correction algorithms. 
For instance, U.S. Patent No. 4,203,671, issued May 20, 1980 to Takahashi et al M discloses a method of detecting 
skin color in an image by identifying pixels falling into an ellipsoid in red, green, blue color space or within an ellipse 

is in two dimensional color space. The problem with this method is that it works well only when an image is properly 
balanced. For an over- or under-exposed image, the technique is not reliable. Furthermore, the technique does not 
work well for those skin colors that deviate from the chosen norm. For example, when the detection method is adjusted 
to detect light Caucasian skin, it fails to properly detect dark skin. 

[0003] There is a need therefore for an improved skin color detection method that avoids the problems noted above. 

20 

SUMMARY OF THE INVENTION 

[0004] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summa- 
rized, according to one aspect of the present invention, a method for detecting skin color in a digital image having 

25 pixels in an RGB color space generally includes the steps of performing statistical analysis of the digital color image 
to determine the mean RGB color values; then, if the mean value of any one of the colors is below a predetermined 
threshold, applying a transformation to the digital image to move skin colors in the image toward a predetermined 
region of the color space; and employing the transformed space to locate the skin color pixels in the digital color image. 
[0005] More specifically, if the mean value of any one of the colors is below a predetermined threshold, a non-linear 

30 transformation is applied to the digital image to move skin colors in the image toward a predetermined region of the 
color space. Then, depending on the preceding step, either the digital image or the transformed digital image is con- 
verted from the RGB space to a generalized RGB space to produce a gRGB digital image; skin color pixels are detected 
within the gRGB digital image; a first skin color image mask is formed based on the detected skin color pixels; a masked 
gRGB image is generated using the first skin color image mask; the masked gRGB image is converted to a hue image; 

35 possible blond hair color pixels are removed from the hue image to produce a modified hue image; a second skin color 
image mask is formed based on the skin color pixels in the modified hue image; if the second skin color image mask 
is smaller than the first skin color image mask by a predetermined amount, then the first skin color image mask is 
selected, otherwise, the second skin color image mask is selected; and finally the selected skin color image mask is 
employed to locate the skin color pixels in the digital color image. 

40 [0006] The advantage of the invention is that it works well even when an image is not properly balanced, and fur- 
thermore works well for a variety of skin colors. 

[0007] These and other aspects, objects, features and advantages of the present invention will be more clearly 
understood and appreciated from a review of the following detailed description of the preferred embodiments and 
appended claims, and by reference to the accompanying drawings. 

45 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0008] 

so Fig. 1 is a schematic diagram of an image processing system useful in practicing the present invention. 

Fig. 2 is a flow chart illustrating the method stages of the present invention. 
Fig. 3 is a detailed flow chart illustrating the skin color detection step shown in Fig. 2. 
Fig. 4 shows an example of an ellipse classifier in two dimensional color space. 

Fig. 5 shows a two dimensional color space having a plurality of skin color classifier regions, as used in the present 
55 invention. 

Fig. 6 shows a two dimensional color space having skin color and blond hair color regions. 

Fig. 7 is a schematic diagram describing the process used to train the ellipse classifier stage shown in Fig. 2. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0009] Because image processing algorithms and systems are well known, the present description will be directed 
in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance 

5 with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing 
and otherwise processing the image signals involved therewith, not specifically shown or described herein may be 
selected from such systems, algorithms, components and elements known in the art. In the following description, a 
preferred embodiment of the present invention will typically be implemented as software or a computer program. Those 
skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Given 

10 the method as described according to the invention in the following materials, software not specifically shown, sug- 
gested or described herein that is useful for implementation of the invention is conventional and within the ordinary 
skill in such arts. 

[0010] Still further, as used herein, the computer program may be stored in a computer readable storage medium, 
which may comprise, for example; magnetic storage media such as a magnetic disk (such as a floppy disk or a hard 
15 drive) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; 
solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any 
other physical device or medium employed to store a computer program. 

[001 1] Referring first to Fig. 1 , an image processing system useful in practicing the present invention includes a color 
digital image source 10, such as a film scanner, digital camera, or digital image storage device such as a compact disc 

20 drive with a Picture CD. The digital image from the digital image source 1 0 is provided to an image processor 1 2, such 
as a programmed personal computer, or digital image processing workstation such as a Sun Sparc 20 workstation. 
The image processor 12 may be connected to a CRT display 14, and an operator interface such as a keyboard 16 and 
a mouse 18. The processed digital images are sent to an output device 20, such a hard copy printer, a long-term image 
storage device, a connection to another processor, or an image telecommunication device connected for example to 

25 the internet. 

[001 2] Referring to Fig. 2, the processing performed according to the present invention by the digital image processor 
1 2 will be described. The digital image, expressed in a device independent RGB color space such as sRGB color space 
is first rank order filtered in a rank order filter step 22. Denote the input RGB image by I = {C,}, where /' = 1 ,2,3 for three 
color-channels, and C, is a matrix with elements of C, {m, n), where m = 0....M - 1 and n = 0....A/ - 1 . The rank-order 
30 filtering is defined as 

(1) 

35 

whereas the set of C,- (m,n) whose value exceeds a predefined threshold T Rank while^is the set of the remaining C ; 
(m,n) of I. This operation is similar to the traditional process of trimmed median filtering. Notice that the purpose of this 
rank-order filtering is not to improve the visual quality of the input image as traditional image processing does; rather, 
it is to reduce the influence of the pixels that have very high intensity values on the subsequent statistics gathering 
40 stage 24. For instance, in situations such as over exposure, or back lit exposure, pixels with values close to 255 will 
be altered or excluded in the mean value calculation. The resultant image of the rank-order filtering can be denoted 
by l R = {c r , c,}, where the elements of c, have the original values, and the elements of c, have the values computed using 
Equation 1. For simplicity, rewrite the rank-order filtered image as l R = {c,;} wherec,=c ; uc, 

[0013] The next step is to compute the color mean-statistics in a statistics step 24 for each color channel using \ R . 
45 This computation produces a set of mean values, that is, {m,-} = mean(\ R )\ where m, = mean(c). A pre-processing 
decision step 26 is made upon evaluating the mean statistics {m,}. If there exits m f < T M , then the pre-processing 
decision is 'Yes', where the threshold T M is an experimentally determined value; for example, for an 8-bit image, a 
value of 100 has been found to be acceptable. 

[0014] If the pre-processing decision is 'Yes', then the process will go to a non-linear processing step 34. The input 
so to the non-linear processing step is the original RGB image I. The non-linear processor employed in this invention is 
color histogram equalization, which is an image processing technique well known to those of skill in this art and thor- 
oughly described in the literature (e.g., see "Digital Image Processing", by Gonzalez and Woods, Addison-Wesley 
Publishing Company, 1992). The input image I is first converted to YIQ space to separate the luminance component 
and chromaticity components using the standard formula: 

55 
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"7] [0.299 0.587 0.114 
/ = 0.569 -0.274 -0.322 
Q 0.211 -0.523 0.312 



(2) 



'3- 



The histogram equalization process is applied to Y only, resulting in an equalized luminance Y". The reverse mapping 
10 from the YIQ to RGB is done by 
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20 [0015] An important issue in color histogram equalization is saturation clipping. Remember that the equalization 
result Y could cause C, to either exceed 255 or becomes less than 0. To prevent this from happening, a clipping 
procedure is used on C. . The pseudo code for clipping saturation is as follows: 



if q>255; A,=255/C,' 

else if C,'<0; Q=0; 

if C;>255; * 2 = 255/q 

else if C;<0; C' 2 =0; 

if C;>255; * 3 =255/q 

else if C;<0; C;=0; 
k = min(k u k 2 ,k 3 ) 



where k,k^ t k 2 and k 3 are initialized to zero. 

40 [0016] If the pre-processing decision is "No", the original image I is sent to the gRGB conversion step 32. 

[0017] In summary, the initialization stage 21 sends an image to a skin-color detection stage 35 depending on the 
pre-processing decision; either the original image I or the histogram-equalized image I' is sent to the gRGB conversion 
step 32. To unify the notation, define \ RGB = {C,}, where / = 1,2,3 for three color-channels, and C ; - is a matrix with 
elements of C,{m,n) t where m = 0,...M-1 and n = 0,.../V-1. Both the original image and the histogram-equalized image 

45 are denoted by \ RGB from now on. 

[0018] Still referring to Figure 2, the input RGB image i RGS is converted in the gRGB Conversion step 32 to a gen- 
eralized RGB image, l gRG0 , which is defined as l gRGS = {c,-}, and c, is a matrix with elements of c,{m,rj), where m = 0,... 
M-1 and n =0,...A/-1. The conversion uses the formula: 



50 

c.(m,n) = £i m ; n) A j e i = [1,2,3]; m -[0..JW -I]; « = [(>,.. JV-1] (5) 



55 



where C, is the individual color channel (R, G, or B) of the input image. This conversion operation is not valid when 
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( 

5 

, and the output will be set to zero. The resultant three new elements are linearly dependent, that is 



j 

so that only two elements are needed to effectively form a new space (gRG plane) that is collapsed from three dimen- 
sions to two dimensions. In most cases, c 1 and c 2 , that is, the generalized R and G, are used in skin color analysis 
15 and skin color detection. The skin colors in the image l gRGB are detected in a skin color detection step 30 that receives 
multiple ellipse vectors from an ellipse classifier 31. The ellipse classifier 31 is trained on a large population of images 
beforehand, which will be discussed below with reference to Fig. 7. The detected skin colors may then be optionally 
segmented in a segmentation step 28. 

[0019] The skin-color detection step 30 is shown in more detail in Figure 3. The generalized RGB image \ gRGB is 
20 first projected on to the gRG plane in the in a projection step 36. This projection produces a gRG image l gRG . The 
projection is affected by simply removing the component c 3 from the gRGB image. The gRG image \ gRG with two 
components c 1 and c 2 is then passed to a first (I) skin color classifier 38 which receives a first ellipse vector (ellipse 
vector I) as shown in Figure 3. Figure 4 illustrates an example of an ellipse 58 in gRG color space 56 that is used to 
designate a region in which skin color pixels are located. The dark region 60 in Figure 4 is the actual skin-color pixel 
25 area. Parameters a, b, (o 1t o 2 ), and 9 are the constants of the ellipse and define the ellipse vector produced by the 
ellipse classifier 31. Parameters a and b are the ellipse axes length, (o 1t o 2 ) is the center coordinates of the ellipse in 
the (c v c 2 ) space 56, and 9 is the angle between the c-, axis and the longer axis of the ellipse. 
[0020] For the first skin-color classifier 38, define an intermediate skin-color mask P, = {Pf(i,j)} M x w- tnat ls - an M x 
N matrix with elements p(i,j). For each pixel of the generalized RG image \ gRG , define an evaluation function E(c 1 (/j'), 
30 c 2 (/j)), where /= 0,-M- 1;y = 0.-/V- 1, c-, (/,/)£ [0,1]; c 2 (/J)e[0,1J. The classification is simply performed as: 



35 



40 



45 



Jl if E{ Ci {i,jlc,{i,j))<\ 
P *> j) = \o else (6> 



where "1" indicates a skin-color pixel and M 0" a non-skin color pixel, and 

H(c 1 (/,y),c 2 (/J))=x^/a 2 + y^ 2 (7) 

and 



x 9 = to (i , j) ~ o x ) cos(0 - 7T 1 4) - (c 2 (i, j) - o 2 ) sin(0 - * / 4) 
y 9 = (c, (i. j) - o x ) sin(0 - n 1 4) - (c 2 (/, j) - o 2 ) cos(0 - n 1 4) 

50 [0021] The intermediate skin-color mask, P /t is passed to an evaluation step 40. The evaluation procedure simply 
computes the ratio, y h of the number of detected skin pixels to the image size. The evaluation step 40 has two branches. 
If y,> T r that is, the evaluation result is 'Good', then the process branches to a skin color region masking step 42. In 
this case, the output from the first skin color classification step 38, P /( is used for masking the l gRGe in the skin color 
region masking step 42. The output of the skin color region masking step 42 is the masked generalized RGB image: 

55 'g/?GB = 'gRGB n P /- ,f Y/ < T r then the evaluat ' on result is ,Not 9 ood '' and tne process branches to a second skin color 
detection classification step 44. While it should be clear that T y may take on a range of different values depending on 
the application, a preferred value for T y is 0.12. 
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[0022] The structure and operation of the second (II) skin color classification step 44 is the same as that of the first 
skin color classification step 38 except it employs a different ellipse vector which determines the position, size and 
orientation of the ellipse in the (c v c 2 ) space. As shown in Figure 5, there are several possible ellipses E1-E7 that can 
be generated by the ellipse classifier 31 . The idea of having multiple ellipses is to reduce false positives while providing 

5 a mechanism for detecting skin colors that deviate from a main skin color region, for example E1 in the (c^,c 2 ) space 
62. As an illustrative case, the use of only two such ellipses (I and II) is described with reference to Fig. 3, but it will 
be understood that a larger number of skin color classifiers and ellipses can be employed in the present invention. 
[0023] There may be cases in which switching from the main skin color classifier (the first classifier step 38) to another 
classifier results in even fewer skin color pixels being detected. Therefore, the results from the skin color classification 

10 steps 38 and 44 are compared in an evaluation and selection step 46, even though the result from the first classifier 
38 was not satisfied in the previous evaluation step 40. In a more general case, the selection decision is made as 
following: 

is y = mdx(y s ) 

if y = ri (9) 
then P = P, 

20 

where P is the first skin-color mask defined as P = {p (ij)} M xN » and p / is the intermediate skin-color mask from Skin 
color Classification I, II and so on if more than two classifiers are used. This first skin-color mask, P, will be used in 
the skin color region masking step 42. 

[0024] If the result from the main skin color detector (the first classifier 38) is satisfied, then P, will be directly used 
25 as the first skin color mask in the subsequent skin color region masking step 42. If not, then the process branches to 
another classifier, the second classifier 44 for example, and both the new result and the result from the main classifier 
(the first classifier 38) will be evaluated and the better (or the best) one will be selected as the first skin color mask to 
be sent to the masking step 42. 

[0025] Notice that there are feedback paths 39 and 41 for the skin color classifiers 38 and 44 respectively in Figure 
30 3. These feedback paths provide the detection result itself to the classifier for evaluation. The evaluation process is to 
compute the ratio of the number of detected skin pixels to the image size. If the ratio is smaller than a predefined 
threshold, the classifier ellipse is then expanded, for instance, by 30% or 40% more, to include more colors. 
[0026] There are two outputs generated from the skin color region masking step 42. One is the masked gRGB image 
and the other one is the skin color mask P itself. These two outputs will be used in the subsequent optional blond-hair- 
35 color pixel removal that is described below. 

[0027] Referring to Fig. 6, we have discovered that blond hair colored pixels 72 occur in the same region where the 
majority of skin colored pixels 71 reside, but next to the skin color region 70 that belong to the people having blond 
hairs. Most currently existing skin detection algorithms do not take the blond hair colored pixels into account. This can 
cause a problem for image understanding applications that are sensitive to the size of the area of skin color, such as 
40 redeye detection algorithms which rely on detecting the actual size of the classified skin color region. A technique for 
removal of blond hair colored pixels is described in the aforementioned copending Serial No. [our Docket No. 
81 321 DMW], entitled "Method of Blond-Hair-Pixel Removal in Image Skin-Color Detection", which is incorporated here- 
in by reference, and briefly summarized below. 

[0028] It has been shown that difficulties arise when dealing with images having faces associated with blond hairs. 
45 in these cases, the conventional skin-color detection process fails to produce satisfied or desired results that would 
give help in redeye detection procedure. It is desirable to have blond-hair-color pixels removed from the masked skin- 
color image obtained by the steps described in the previous sections. 

[0029] However, it is not a trivial task to parameterize the sub-regions such as the blond hair color region 72 and the 
skin color region 70 in the (c 1t c 2 ) space 68 so that the hair color can be separated from the face skin color. If the space 
50 dimension drops down to one, the separation of blond hair color pixels from the skin color pixels becomes fairly easy. 
This further reduction of dimension size is realized by converting the masked generalized RGB image \ gRGB to a hue 
image H = {h(m,n)} MxN in a converting to hue image step 54. A typical hue conversion is performed as: 



55 



6 



EP 1 223 550 A2 



10 



// c, (m,/z) = c max <fc n) = (c 2 - c 3 (m, /i))/^ - c min ) 
efrez/ c 2 = c max rfo = 2 + (c 3 - c, (m,/i))/(c raax - c roin ) ( 10 ) 

e/se//* c 3 = do h(m,n) = 4 -h (c, (m,n) - c 2 (m,*))/^™ - c min ); 

A(7n,n) = A(^/i)*60°; 
if h(m i n)< 0.0 Jo = A(/h,/i) + 360°; 



In a blond-hair color detection step 52, a predefined partition parameter T H =15 is used to determine if an element h 
(m,n) is a skin pixel or a blond hair pixel. A second mask,p= {pi{ij)} M xN> is formed. If h(m,n) is a skin pixel, then the 
corresponding element p(m,n) = 1, elsep(m, n) = 0. In some cases, the blond-hair-color pixel removal may take away 

15 true skin-color pixels and the resultant second skin-color mask shrinks to an unusable small region. Therefore, the first 
skin-color mask P will be called back and the second the skin-color mask? is discarded. This action is performed in an 
evaluation and selection step 50 following the blond-hair color detection step 52 as shown in Figure 3. If the second 
skin color image mask is smaller than the first skin color mask by a predetermined amount, the first skin color image 
mask is selected, otherwise, the second skin color image mask is selected. The masked RGB color image is the 

20 result of an AND operation of the selected skin color image mask and the original RGB color image l RGS . This operation 
is performed in a skin color region masking step 48. 

[0030] Referring to Figure 7, a large image pool 74 containing over a thousand images from different sources is 
prepared for sample skin color patch collecting. It is not necessary to collect skin patches from all prepared images. 
A set of randomly selected images should be representative enough for the whole image pool, in theory and in practice 

25 as well. A set of uniformly distributed indices is used to retrieve the images from the image pool and form a set of 
random selected images 76. Sample skin color patches 78 are collected manually from the retrieved images. The 
collected skin color patches are then converted from the RGB space to the gRGB space 80 using Equation (5) above. 
The gRG data of the gRGB image is then evaluated in term of its distribution density 82 in the gRG plane 84, that is, 
the (c 1( c 2 ) space as described above. In practice, the distribution density is approximated with the histogram of the 

30 gRG data. The distribution density is further projected onto the (c v c 2 ) space after eliminating some of its elements 
whose height is less than 5% of the distribution density peak. The projected cluster 88 forms approximately an ellipse 
region in the gRG plane 86. The final step of the ellipse classifier training is to find the parameters for the ellipse region. 
The best-fit ellipse 92 in the gRG plane 90 is computed on the basis of moments. An ellipse is defined by its center 
(Ot ,o 2 ), its orientation e and its minor axis a and major axis b (see Figure 4). The center of the ellipse region is computed 

35 by 

40 °2 = m 0\ 1 m 0O 

where the moments are computed as: 



45 ao oo 

m pg= j jcfcinc^cjdc^c, (12) 

-eo-co 

50 where f(c v c 2 ) = 1 1 in this application. The orientation 0 can be computed by determined the least moment of inertia: 



55 



{M20-M02J 



0 = O.5arctan ^" (13) 
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where the central moments are calculated as: 



M m = J Jfe -o$f{.c x ,cJdc x dc x (14) 

-OO-oO 

and finally the length of minor and major axis can be computed as: 

10 

a = l.0623(A 3 /Bf US 
b = l.Q623(B 3 /Af m 
is A= JJi^-oJsmG-ici-oJcose'f (15) 

(ci ,c 2 )eellipse _ region 

B = Xt( c i -^) cos ^~( c 2 -o 2 )sm&Y 

fa ,c 2 }=eUipse_region 

20 

The above computation provides an initial set of ellipse vector [a,b,o v o 2 £] 94. In practice, manual adjustment is needed 
to best fit the final ellipse to the gRG data cluster 88. 

[0031] The subject matter of the present invention relates to digital image understanding technology, which is un- 
derstood to mean technology that digitally processes a digital image to recognize and thereby assign useful meaning 
25 to human understandable objects, attributes or conditions and then to utilize the results obtained in the further process- 
ing of the digital image. 

Claims 

30 ■ 

1. A method for detecting skin color in a digital image having pixels in an RGB color space, comprising the steps of: 

a) performing statistical analysis of the digital color image to determine the mean RGB color values; 

b) if the mean value of any one of the colors is below a predetermined threshold, applying a transformation to 
35 the digital image to move skin colors in the image toward a predetermined region of the color space; and 

c) employing the transformed space to locate the skin color pixels in the digital color image. 

2. The method as claimed in claim 1 wherein step (c) further comprises the steps of: 

40 converting the transformed digital image from the RGB space to a generalized RGB space, to produce a gRGB 

digital image; 

detecting skin color pixels within the gRGB digital image; 

forming a first skin color image mask based on the detected skin color pixels; and 

employing the first skin color image mask to locate the skin color pixels in the digital color image. 

45 

3. The method as claimed in claim 2 wherein step (c) further comprises: 

forming a second skin color image mask based on the detected skin color pixels; 

selecting either the first skin color image mask or the second skin color image mask based on predetermined 
so criteria; and 

employing the selected skin color image mask to locate the skin color pixels in the digital color image. 

4. The method as claimed in claim 2 wherein step (c) further comprises the steps of: 

55 generating a masked gRGB image using the first skin color image mask; 

converting the masked gRGB image to a hue image; 

removing possible blond hair color pixels from the hue image to produce a modified hue image; 
forming a second skin color image mask based on the skin color pixels in the modified hue image; 
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if the second skin color image mask is smaller than the first skin color image mask by a predetermined amount, 
selecting the first skin color image mask, otherwise, selecting the second skin color image mask; and 
employing the selected skin color image mask to locate the skin color pixels in the digital color image. 

A computer program product for performing the method as claimed in claim 1 . 
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